Quantifying the quality of peer reviewers through Zipf's law

摘要

This paper introduces a statistical and other analysis of peer reviewers inorder to approach their "quality" through some quantification measure, therebyleading to some quality metrics. Peer reviewer reports for the Journal of theSerbian Chemical Society are examined. The text of each report has first to beadapted to word counting software in order to avoid jargon inducing confusionwhen searching for the word frequency: e.g. C must be distinguished, dependingif it means Carbon or Celsius, etc. Thus, every report has to be carefully"rewritten". Thereafter, the quantity, variety and distribution of words areexamined in each report and compared to the whole set. Two separate months,according when reports came in, are distinguished to observe any possiblehidden spurious effects. Coherence is found. An empirical distribution issearched for through a Zipf-Pareto rank-size law. It is observed that peerreview reports are very far from usual texts in this respect. Deviations fromthe usual (first) Zipf's law are discussed. A theoretical suggestion for the"best (or worst) report" and by extension "good (or bad) reviewer", within thiscontext, is provided from an entropy argument, through the concept of "distanceto average" behavior. Another entropy-based measure also allows to measure thejournal reviews (whence reviewers) for further comparison with other journalsthrough their own reviewer reports.